OPEN 3 ACCESS Freely available online 



•0-PLOS I o-^E 



Accuracy of the Interpretation of Chest Radiographs for (a\ 
the Diagnosis of Paediatric Pneumon 

Mohamed A. Elemraid^'^% Michelle Muller^ David A. Spencer'*, Stephen P. Rushton^ Russell Gorton^ 
Matthew F. Thomas'*'^, Katherine M. Eastham^, Fiona Hampton", Andrew R. Gennery^'^, Julia E. Clark^, 
on behalf of the North East of England Paediatric Respiratory Infection Study Group 

1 Department of Paediatric Infectious Disease and Immunology, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, United Kingdom, 2 Institute 
of Cellular Medicine, Newcastle University, Newcastle upon Tyne, United Kingdom, 3 Department of Radiology, Newcastle upon Tyne Hospitals NHS Foundation Trust, 
Newcastle upon Tyne, United Kingdom, 4 Department of Respiratory Paediatrics, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, United 
Kingdom, 5 Biological, Clinical and Environmental Systems Modelling Group, School of Biology, Newcastle University, Newcastle upon Tyne, United Kingdom, 6 Regional 
Epidemiology Unit, Public Health England North East, Newcastle upon Tyne, United Kingdom, 7 Department of Paediatrics, Sunderland Royal Hospital, Sunderland, United 
Kingdom, 8 Department of Paediatrics, James Cook University Hospital, Middlesbrough, United Kingdom, 9 Department of Paediatric Infectious Disease, Royal Children's 
Hospital, Brisbane, Queensland, Australia 



Abstract 

Introduction: World Health Organization (WHO) radiological classification remains an innportant entry criterion In 
epidemiological studies of pneumonia in children. We report Inter-observer variability In the Interpretation of 169 chest 
radiographs In children suspected of having pneumonia. 

Methods: An 18-month prospective aetlological study of pneumonia was undertaken In Northern England. Chest 
radiographs were performed on eligible children aged <16 years with clinical features of pneumonia. The Initial radiology 
report was compared with a subsequent assessment by a consultant cardlothoracic radiologist. Chest radiographic changes 
were categorised according to the WHO classification. 

/fe5i//f5; There was significant disagreement (22%) between the first and second reports (kappa = 0.70, P<0.001), notably in 
those aged <5 years (26%, kappa = 0.66, P<0.001). The most frequent sources of disagreement were the reporting of 
patchy and perihilar changes. 

Conclusion: This substantial Inter-observer variability highlights the need for experts from different countries to create a 
consensus to review the radiological definition of pneumonia In children. 
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Introduction 

Chest radiograph is frequently performed when managing 
pneumonia in children [I], but usually does not affect the chnical 
outcome [2]. hi epidemiological studies, the chest radiograph 
remains a major criterion in classifying pneumonia [3,4]. 
However, variability in its interpretation for the diagnosis of 
pneumonia in children is a recognised problem It has been 
suggested that if radiologists foUow the standardised World Health 
Organization (WHO) radiological definitions of pneumonia [3] , 
this would allow more accurate comparative data in epidemio- 
logical studies for assessment of the impact of pneumococcal 
vaccination [4]. Broadly four categories are defined: '^End-point 
consolidation", "Other (non-end-point) infiltrate", "Pleural effu- 
sion" and "No pneumonia" (Table 1). 

We conducted a study to explore the effect of the implemen- 
tation of pneumococcal conjugate vaccine on the aetiology of 



childhood community-acquired pneumonia (CAP) [6]. Radiolog- 
ical findings were part of the study entry criteria. The aim of this 
analysis was to characterise inter-observer variability in the 
interpretation of chest radiographs for the diagnosis of pneumonia 
in children according to the WHO radiological classification [3]. 

Methods 

Study Design and Participants 

A prospective study to investigate the aetiology of CAP in 
children was undertaken from October 2009 to March 2011 in 
two teaching centres in North of England; the Newcastle Hospitals 
and South Tees Hospitals NHS Foundation Trusts [6] . Research 
teams of doctors and nurses led and ascertained the standardised 
diagnosis of pneumonia and the recruitment procedures across the 
two study centres. In the UK, children are assessed by a General 
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Table 1. Summary of the WHO definitions of reporting chest radiographs in children with pneumonia [3]. 



1 . "End-point consolidation": a dense opacity that may be a fluffy consolidation of a portion or whole of a lobe or of the entire lung, often containing air bronchogram 
and sometimes associated with pleural effusion. 

2. "Other (non-end-point) infiltrate": a linear and patchy densities (interstitial infiltrate) in a lacy pattern involving both lungs, featuring peribronchial thickening and 
multiple areas of atelectasis with lung inflation is being normal to increased. It also includes minor patchy infiltrates that are not of sufficient magnitude to constitute 
primary end-point consolidation, and small areas of atelectasis which in children can be difficult to distinguish from consolidation. 

3. "Pleural effusion": this refers to the presence of fluid in the pleural space between the lung and chest wall. Mostly this will be seen at the costo-phrenic angle or as a 
layer of fluid adjacent to the lateral chest wall. This does not include fluid seen in the horizontal or oblique fissures. Pleural effusion is considered as primary end-point if 
it is in the lateral pleural space (and not just in the minor or oblique fissure) and is spatially associated with a pulmonary parenchymal infiltrate (including other 
infiltrate), or if the effusion obliterates enough the hemithorax to obscure an opacity. 

4. "No pneumonia": if there is no evidence of consolidation, infiltrate, or pleural effusion. 



doi:l 0.1 371 /journal.pone.Ol 06051 .tool 

Practitioner in primary care or accident and emergency team and 
then referred to a iiospital-based paediatrician if secondary care is 
required. 

Informed written consent was obtained from parents as well as 
assent from older children. Caldicott approval was granted and the 
study along with the informed consent procedures were ethically 
approved by the Newcastle and North Tyneside Research Ethics 
Committee (No: 08/H0906/105), and the Research Approval 
Board at South Tees Hospitals NHS Foundation Trust (No: 
2008075). 

Enrolled children aged ^16 years who were presented to 
paediatric services with features suggestive of lower respiratory 
tract infection including any of fever, tachypnoea, dyspnoea, 
cough, respiratory distress and auscultatory chest crackles, with 
chest radiographic findings consistent with pneumonia as deter- 
mined initially by the admitting paediatrician. Paediatricians were 
not asked to give specific radiological interpretations which were 
provided by radiologists. All children irrespective of the radiolog- 
ical findings received treatment for pneumonia according to the 
British Thoracic Society guidelines [7] . 

As this study was on the CAP aetiology, exclusions included 
clinical bronchiolitis or hospitalization in the preceding three 
weeks. Children with recent hospitalization were excluded in order 
to eliminate the potential risk of having hospital-acquired 
pneumonia rather than CAP. Children with underlying chronic 
chest diseases (such as cystic fibrosis) were also excluded to avoid 
any ambiguity in the interpretation of acute and chronic changes 
on chest radiographs. 

Laboratory Procedures 

Microbiological and vrrological testing informed the aetiology of 
pneumonia which was previously published [6]. Identified 
pathogens were categorised as viral, bacterial or mixed viral- 
bacterial infections according to defined diagnostic criteria 
(Table 2) [6]. 

Radiology 

All chest radiographs were digitally taken, either with a flat 
panel detector or with a digital storage system. They were first 
reported by consultant radiologists locally as per routine clinical 
care and viewed electronically via the Picture Archiving and 
Communications System (PACS). There were uniform and regular 
quality assessments performed on the system performance 
including display characteristics. All reporters used similar 
workstations of radiological standards when reporting the chest 
radiographs. The location of the chest radiographic changes was 
documented by radiologists on every report and used for 
variability comparisons. 



Using the full text written first reports, each radiograph was 
categorised into lobar (end-point consolidation), patchy, perihUar 
(non-end-point consolidation/ infiltrate) or normal (no pneumonia) 
according to the WHO criteria [3,4]. Effusion with fluid in the 
pleural space between the lung and chest wall was considered as 
primary end-point and classified simply as either present or absent 
[4] . This does not include fluid in the horizontal or oblique fissures 
as defined in the WHO radiological classification [4]. First reports 
were generated with the benefit of clinical information, a standard 
institutional requirement for routine reporting. All radiographs 
were reviewed by a second senior consultant cardiothoracic 
radiologist (MM) at the regional centre who was blinded to the first 
report. MM works primarily in paediatric radiology and regularly 
reports chest radiographs for children with cardio-respiratory 
diseases including pneumonia. Radiologists involved in performing 
the first and second reporting received the same training in 
radiology including the classification of radiological pneumonia. 

A workshop including MAE, MM, DAS and JEC was carried 
out before the application of WHO criteria [3] on the first reports 
and performing the second reading in order to discuss and refine 
the potential definitions which could be a source of disagreement 
such as interstitial infiltrates of patchy or perihilar changes. There 
was a consensus agreement among the study team that if more 
than one radiographic change were reported, then in line with 
WHO recommendations the most significant one is reported [3]. 
The WHO criteria were prioritised according to the clinical 
significance, as follows: lobar (end-point consolidation) in favour of 
other changes (non-end-point infiltrates) if both were present [3] . 
There was no ambiguity on the wording of first reports that might 
cause confusion on categorization. 

Statistical Analysis 

Inter-observer variability in the interpretation of chest radio- 
graphs was measured by the comparison of first reports with their 
second reading. Data analysis was performed using the PASW 
Statistics 19 program. The significance of inter-observer variability 
was assessed using fisher's exact test because there were small 
values <5. Cohen's kappa index (k) was calculated to measure the 
agreement between the first and second readers above that which 
would be expected by chance. 

Results 

A total of 169 children were identified and treated for 
pneumonia and/or empyema (53% males, 73% aged <5 years, 
mean age 3.8±3.72 years, and age range from 0.05 to 16.7 years). 
Of those, 46 had chest radiograph reported as normal on the first 
reports, but on the second reading six (13%) had abnormal 
changes (i.e. false negative); four lobar and two patchy. AH of the 
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Table 2. Laboratory investigations 


and diagnostic criteria of likely causative pathogens of pneumonia. 




Sample 


Pathogen/antigen 


Tests 


Interpretation 


Serum 


Respiratory viruses 


Complement fixation 


Acute titre > 1/1 28 or 4-fold 
rise between paired sera 


Atypical bacteria 




Mycoplasma 


IgM antibody 


Positive 




Group A 
Streptococcus 


Antistreptolysin 0 titre (lU/mL) 


Acute 2-fold rise or 4-fold 
rise between paired sera 


Blood 


Bacteria 


Culture 


Growth 




Streptococcus 
pneumoniae 


Real-time PCR 


Positive 


Nasopharyngeal 
secretions/sputum 


Respiratory 
viruses 


Real-time PCR 


Positive 


Tracheobronchial secretions 
(collected via endotracheal 
tube or bronchoalveolar lavage} 


Respiratory 
viruses 


Real-time PCR 


Positive 




Bacteria 


Culture/Real-time PCR 


Growth/Positive 


Pleural fluids 


Bacteria 


Culture 


Growth 




Pneumococcal 
antigen 


ELISA^ 


Positive 




Streptococcus 
pneumoniae 


Real-time PCR 


Positive 



^ELISA, enzyme-linked immunosorbent assay. 
doi:l 0.1 371 /journal.pone.Ol 06051 .t002 



false negative cases received antibiotic treatment (median, 7 days), 
and none developed any complication. Fourteen (11.4%) were 
initially reported as having radiological changes, were reported as 
normal radiographs on the second review (i.e. false positive) 
(Table 3). 

AH radiologists agreed that all chest radiographs were suitable 
for interpretation. There was significant inter-observer variability 
in the interpretation of chest radiographs (A = 0.70, P<0.001), 
with patchy (48.8%) and perihUar (28. 1 %) changes being the main 
components of this variability (Table 3). Levels of disagreement 
were highest among children aged <5 years compared to those 
aged >5 years (26%, A = 0.66 versus 11%, A = 0.83, P<0.001). 
There was no disagreement on reporting lobar findings in the <5 
years age group, disagreement was mainly related to patchy and 
perihUar changes. 

Pleural effusion was present at first reading of the films in 10% 
(17/169) compared to 22% (37/169) on review. Variation in 
reporting of pleural elfusion was 11.8% (A = 0.57, P<0.001). 
However, if the presence of a pleural effusion was reported in the 
first report there was no disagreement about this in the second 



report. In contrast 13.2% of pleural effusions were reported only 
on the second report and not in the first report. 

Discussion 

We found substantial inter-observer variability in the interpre- 
tation of chest radiographs for the diagnosis of paediatric 
pneumonia. This has been recognized since radiology reporting 
was initiated in the middle of last century [8,9], and continues 
despite the acceptance of the recommended WHO criteria for 
reporting chest radiographs of pneumonia in children [3,4]. 

Yet, subtle radiographic changes can be difficult to recognise or 
interpret [10]. The initial interpretation of chest radiographs is 
usually performed by clinicians with the radiologists' reports 
following later, often after the patient has been discharged from 
hospital [11]. Interpretation by clinicians could be biased by 
inadequate training in radiology and lack of clinical information 
may limit the accuracy of reporting by the radiologists [12]. For 
research purposes blinded interpretation of the chest radiograph 
may improve detection of subtle changes and differentiating 



Table 3. Inter-observer variability and agreement in the chest radiographs reporting. 



First reading Second reading (gold standard) Disagreement^ 



Radiograpliic cKianges 


n (%) 


Lobar 


Patchy 


Perihilar 


Normal 


n (%) 


Lobar 


48 (28.4) 


47 


1 


0 


0 


1 (2.1) 


Patchy 


43 (25.4) 


7 


22 


5 


9 


21 (48.8) 


Perihilar 


32 (19.0) 


4 


0 


23 


5 


9 (28.1) 


Normal 


46 (27.2) 


4 


2 


0 


40 


6 (13.0) 


Total 


169 


62 


25 


28 


54 


37 (22.0) 



Visher's exact test, P<0.001; Kappa = 0.70 (proportion of subjects on which readers would be expected to agree). 
doi:l 0.1 371/journal.pone.Ol 06051 .t003 
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normal biological variants [13]. Making clinical information 
available may reduce inter-observer variability but does not result 
in marked improvement in the overall accuracy [14]. 

This study shows that most inter-observer variability is related 
to the interpretation of patchy and perihUar changes, which need 
careful viewing and the availability of clinical information to 
facilitate their reading [15]. It is well recognised that abnormal 
chest radiographs may be interpreted as normal [15], but 
surprisingly four of the normal reports had lobar changes on 
review. Similarly, 13% had a previously undetected pleural 
effusion. The variation in reporting of chest radiographs for those 
aged <5 years confirms the particular challenge of making a 
radiological diagnosis of pneumonia in this age group [10,16]. The 
findings in chest radiographs reported according to the WHO 
radiological classification of Pakistani children aged 2-59 months 
diagnosed with non-severe pneumonia showed normal films in 
82% (1519/1848) and lobar consolidation in 26 children [17]. It is 
widely accepted in the literature that chest radiographs cannot 
reUably differentiate viral from bacterial aetiology of pneumonia 
[7,18]. Therefore these variations on their interpretation do not 
significandy affect the clinical outcomes and management 
decisions of pneumonia in children [2,7,18]. 

Despite the specialized training in paediatric radiology and 
advanced technology, human error remains a likely factor [9] . The 
rate of false negative reports between the two interpretations of 
chest radiographs is a well-recognized problem [15]. This may 
jeopardize the results of epidemiological studies by underestimat- 
ing the true burden of pneumococcal pneumonia [19]. Madhi and 
Klugman [19] raised a concern that radiologically-defined 
pneumonia according to the WHO criteria [3] may underestimate 
the actual effect of pneumococcal conjugate vaccine in preventing 
pneumococcal pneumonia by up to 63% when data from vaccine 
trial in South Africa were further analysed [19]. They suggested to 
include non-specific infiltrates that are associated with C-reactive 
protein level >40 mg/L when evaluating the vaccine impact [19]. 

In previous pneumococcal vaccine efficacy studies the radio- 
graphic evidence of pneumonia was observed in up to 34% of the 
enrolled children [20]. Despite the application of the WHO 
criteria [4], the concordance rate between two trained reviewers 
was only 48% (250/521) [21]. The degree of variabUity of 
reporting chest radiographs from the present study demonstrates 
that methodological diffcrciux-s are still a problem in the 
epidemiological studies of pneumonia in children. The WHO 
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